Item Pool Construction Using Mixed Integer Quadratic Programming (MIQP)

This study uses mixed integer quadratic programming (MIQP) to construct multiple highly equivalent item pools simultaneously, and compares the results from mixed integer programming (MIP). Three different MIP/MIQP models were implemented and evaluated using real CAT item pool data with 23 different content areas and a goal of equal information functions across pools and within each content area. The study addresses two important practical questions: (a) how many evaluation points should the objective functions of the MIP/MIQP models use when the targets have numerous non~N(0,1) distributions, and (b) how should the solver be structured when an item bank is gigantic? The study finds that all three MIP/MIQP models could be used effectively to construct highly parallel item pools and content bins when five evaluation points were used. Utilization of these techniques can replace current laborious manual pool construction methods. Introduction For long-term quality control of computerized adaptive testing (CAT) programs, it is crucial to construct and maintain quality item pools that are consistent over time in terms of their psychometric properties and their match to the ability distributions of the test takers. Regardless of the adaptive algorithm, consistency in pool quality is a necessary condition for consistency in the score accuracy for test takers. Construction of multiple parallel item pools is often challenging, however, because of the number of factors to be considered (e.g., bank information, content balancing, exposure rate, response time, etc.) and the limited number of items available in the item bank. In applied settings, the goal is often to maintain consistency in pool information functions across pools and to balance content in terms of the number of items within each content area in each pool. Item pool construction is usually performed manually using sampling techniques. Constructing multiple parallel item pools that meet all the pool specifications by hand, however, is very labor intensive, especially when there are numerous content constraints and the number of item pools to be constructed and/or the number of items for each pool is large. This study investigates the feasibility of using mixed integer programming (MIP) and mixed integer quadratic programming (MIQP) to construct multiple highly equivalent item pools that meet content, exposure, and psychometric constraints, including the goal that each content area should have the equivalent information functions across pools. The study explores three models using three different evaluation points in objective functions. The quality of the approaches is evaluated in terms of pool information function consistency and performance of the solver under each condition. P.O. Box 2969 • Reston, Virginia • 20195 • USA • gmac.com • mba.com Item Pool Construction Using MIQP, Han & Rudner Mixed Integer Programming Models Various industries, from delivery services to financial institutions, have made wide use of linear programming (LP) models to optimize resources while maximizing outcomes. In the 1980s, the educational measurement field began adopting LP models for optimal test design for applications of automated test assembly (ATA; van der Linden, 2005). For ATA, the most common approach to LP is to introduce as many 0–1 binary variables as there are items in the bank, and then to let the solver software identify an optimal test design by finding the best combination of binary variables that will yield the maximum (or minimum depending on a problem) objective value (Theunissen, 1985). When multiple test forms are constructed at the same time, the LP often takes the form of network-flow programming, in which an array of integer variables (i × j) is determined in order to optimize the flows between i supply nodes and j demand nodes (Armstrong, Jones, & Wang, 1995; van der Linden, 1998). Because the decision variables of this LP model are integers, it is generally known as a mixed integer programming (MIP) model, and will be referred to as such in this paper. Item pool construction, the problem addressed in this paper, is not technically very different from an automated test assembly. Just as MIP models for ATA can be used to assemble the best sets of tests out of a given item pool, the MIP model can, in theory, also systematically assemble optimal sets of item pools. Ariel, Veldkamp, and van der Linden (2004) used the MIP model to optimally divide an item bank into multiple operational pools with similar content distributions. Van der Linden, Ariel, and Veldkamp (2006) discussed the formation of pools to meet the ability distributions of the targeted test takers while meeting content constraints. In practice, meeting content constraints, often a difficult task in itself, does not necessarily mean that the collection of test questions within a content area will meet the important goal of having similar information functions across pools. For CAT pool construction, whether done by hand or computer, it is more common to minimize the difference between an actually constructed item pool and a target, which often is determined by the examinee’s proficiency distribution. This can be modeled by evaluating the information function conditioned on θ: Minimize 1 [ ( | ) ] J j j I j x d θ θ θ τ θ ∞